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Abstract: This paper presents an effort to revitalize a large introductory engineering course 
for incoming freshman students that teaches them analytical design through a project-based 
curriculum. This course was completely transformed from a seminar-based to a project-based 
course that integrates hands-on experimentation with analytical work. The project is centered 
on a reconfigurable trebuchet kit that student groups assemble and work to identify design 
decisions that will maximize projectile launch distance. Challenges include streamlining the project 
experience for the large enrollment (up to 148 students) with limited contact hours, and helping 
students fuse hands-on experiences with quantitative engineering analysis. A mixed-methods 
approach supported the claim that the curriculum improved the students' engineering judgment 
and demonstrated to students the value of engineering analysis and mathematical models in 
practical engineering design. A rigorous statistical analysis of student trebuchet launch performance 
at different course stages is included. A qualitative assessment of student learning is derived 
through students' reflection of their course experience. Comprehensive results comparing students' 
design iterations versus algorithmic design optimization iterations provide important insights into 
student design intuition, paving the way for hybrid design education models that teach students 
how to combine human design intuition with quantitative design tools to design superior systems. 

Keywords: project-based learning; engineering judgment; design education; engineering education; 
trebuchets; reconfigurable kits; interactive computer models; model-based design 


1. Introduction 

Traditional engineering design courses often have relied on happenstance to link theory with 
practice. Noticing these deficits in student practical engineering design experience, the Department 
of Industrial and Enterprise Systems Engineering at University of Illinois at Urbana-Champaign 
decided to revamp its freshman-level engineering course by introducing a project-based curriculum. 
The premise was that a project-based, hands-on activity that was connected carefully with 
quantitative engineering analysis would introduce students to the rigors of engineering design 
practice at an early stage in their engineering career, along with other benefits. 
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1.1. Project-Based Learning via Trebuchets 

Project-Based Learning (PBL) is a teaching method in which students gain knowledge and skills 
by working for an extended period of time to investigate and respond to an engaging and complex 
question, problem, or challenge [1]. Simply put, project-based learning conveys learning through 
experiences [2]. PBL curricula are designed in a way in which students work in groups to solve 
challenging issues that are authentic in nature but based within a curriculum [3,4]. 

Over the past several years, the United States has experienced a shortage of qualified engineers. 
Being cognizant of this fact, the U.S. National Science Foundation (NSF) called for reform in 
engineering education [5]. This call for reform included and emphasized project-based learning. 
PBL encourages self-directed learning that in turn supports life-long learning (which links into 
the department's overarching goal). That ultimate goal is to provide students with the required 
knowledge base and necessary skill set, while also helping students develop attitudes that support 
becoming efficient and effective engineers. 

The PBL authentic approach [6] was introduced to the curriculum via a reconfigurable trebuchet 
kit. A trebuchet is a machine that converts gravitational potential energy to kinetic energy to launch 
a projectile. These kits and associated curriculum were designed to enhance student engagement, 
participation, and motivation, which are keys to linking theory to practice in engineering analysis 
and design. Students were presented with situations that required them to become active learners 
rather than relying on the rote methods of lecture [7]. Research has found that implementation 
of project-based lab activities into engineering design courses help bridge the gap between theory 
and practice [6]. Other available research suggests that PBL courses appear to improve retention, 
student satisfaction, diversity, student learning [8], and provide beneficial 'soft skill' development for 
students [9]. 

Many PBL activities employ open-ended projects that are especially helpful in aiding 
development of synthesis skills and creative idea generation and exploration. When these 
open-ended PBL projects include a significant design element they are sometime referred to as 
design-based learning (DBL). Thus, DBL is a type of project-based learning which involves students 
engaged in the process of developing, building, and evaluating a product they have designed [10]. 
Open-ended projects, however, often are time and resource intensive, may require significant 
prerequisite coursework to obtain the necessary tools, and often are difficult to connect with rigorous 
engineering analysis except in capstone or graduate-level projects. More structured projects support 
streamlined learning activities that are less time and resource intensive but are appropriate for 
learning objectives that are different from open-ended projects (including integration of analysis). 
For example, open-ended engineering design projects support learning important elements of design 
processes, such as problem identification and formulation, design concept generation, trade-off 
negotiation, and project management. Many argue design is fundamentally open ended [11]. 
In cases where students have sufficient technical preparation (e.g., upper-division engineering 
analysis courses) and ample time is available, open-ended design projects may also provide students 
with opportunities to learn the important connection between engineering design practice and 
rigorous engineering analysis. The specific course addressed in this article, however, is a first-year 
undergraduate engineering course that meets for only one hour per week. Time limitations and 
lack of deep technical background constrain PBL activities for this type of course. Discovering 
how engineering analysis supports engineering design practice, however, provides vital context for 
first-year undergraduates as they begin a sequence of challenging courses in fundamental science 
and engineering analysis. Given these constraints, how then can educators help beginning students 
experience the relationship between analysis and engineering practice? Many first-year PBL courses 
neglect rigorous analytic design in favor of more tangible activities on open-ended design problems 
(differentiating our approach from classical DBL activities). While these strategies produce valuable 
learning outcomes, a need exists for integrating analysis into design learning at early stages of 
education. Our hypothesis here is that a structured PBL curriculum with targeted learning outcomes 
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provides an efficient and engaging learning strategy that addresses the interface between engineering 
analysis and practice. 

The pedagogical approach proposed here aims to help students develop a specific understanding 
of the synergistic relationship between engineering analysis and practical engineering knowledge 
and intuition. In the simplest terms, intuition may correspond to simplified mental models or 
heuristics. It is helpful for an engineer to have good mental models based on experience and practical 
understanding of engineering systems, but it is also important for them to understand the limitations 
of these mental models [12]. Relying solely on intuitive mental models can lead to design fixation 
especially in case of inexperienced engineers where they may intuitively hold false assumptions about 
the system being designed, abide by nonexistent limitations, feel overwhelmed or have incomplete 
or partial information [13]. The design fixation can be mitigated through design exploration based 
on rigorous analysis or experimentation which can then feed back into strengthening these mental 
models and making them more sophisticated or accurate [14-16]. One such rigorous strategy is 
continuous engineering design optimization, which is an important design methodology for many 
engineering applications [17]. Introducing students to both intuition-based and quantitative design 
decision strategies is important, especially early in their engineering education. 

The integration of engineering intuition and analysis is referred to in this work as engineering 
judgment. Although there is no clear definition for engineering judgment in the literature, our 
definition is based on the approach proposed in Ref. [12], Here we define engineering judgment as the 
ability to utilize simultaneously practical engineering intuition and quantitative engineering analysis 
to support improved engineering decisions. A primary learning objective is to provide first-year 
students with a solid foundation for the development of their own engineering judgment, which 
will be built upon though later coursework. An additional objective is to provide students with a 
proper perspective on how content in upcoming courses, focused on fundamentals and quantitative 
analytical tools, relates to finding solutions to real engineering problems. A lean curriculum was 
developed with targeted learning activities centered on a structured design project. This project 
involves trebuchet kits that are assembled by students, and that can be reconfigured to change 
dynamic behavior in complicated ways. Reconfigurable capability provides an opportunity for 
students to make design decisions without the need to construct one or more physical prototypes 
from scratch. While there is certainly value in open-ended design and fabrication activities, these 
are provided in other courses. The focus here is instead on making design decisions supported by 
quantitative engineering analysis. Students learn to improve their designs through physics-based 
understanding of the system, and through rational application of modern simulation-based tools. 
The use of trebuchets in classrooms is certainly not a novel idea. Using trebuchets as a means to 
understand the relationship between challenging practical design decisions and rigorous analytical 
tools, however, is. 

1.2. Previous Pedagogical Use of Trebuchets 

Trebuchets or other projectile-throwing, simple machines are commonly used by educators 
in PBL activities or for demonstrations. The accessibility of trebuchets permits activities where 
students at a variety of levels can engage (even beyond engineering and science related courses [18]). 
Engineering upperclassmen [18-20], engineering freshman [21-23], high school students [24,25], 
middle school students [26], and even K-5 students (as the authors have personally experienced) 
are all levels of education where trebuchet-based learning activities have played some part. 

Many of the reported activities have curricula developed around different learning objectives 
and trebuchet kits/models not suitable for analytical design activities. Open-ended trebuchet 
activities and rigorous analytical design exploration do not need to be mutually exclusive activities; 
however, much of the students' time during open-ended activities tends to be occupied with 
important, creative tasks (which are more qualitative in nature) than quantitative exploration of 
their proposed trebuchet design [22,24,26]. Since teaching analytical design methods is not one of 
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the primary goals of these activities, it is not surprising that only a limited amount of time is spent 
modeling and perfecting their designs. 

Computer models of trebuchets [20,25] facilitate rapid design iteration through model-based 
design strategies. Using mathematical optimization tools in conjunction with computer models to 
identify high-performance designs is an especially powerful strategy [20,27]. For these efforts to 
correspond to actual trebuchet design problems, the models and design problems must be based on 
realistically tunable design variables. For example, adjusting the projectile release angle is convenient 
from a modeling perspective, but this cannot be adjusted directly on a physical trebuchet. Realistically 
tunable parameters that influence release angle should be chosen instead as design variables. In 
addition, computer models can be used jointly with physical trebuchets to demonstrate the value of 
model-based design when compared to design exploration through physical testing [20,25]. 

The variety of trebuchet-based learning activities were shown to achieve a variety of important 
outcomes, including helping students to connect theoretical course material to realistic engineering 
design problems [21], improving inclination towards continuing in a STEM field [24], and improving 
problem-solving skills [19]. Here we build on the efforts described in the above literature through the 
development of a trebuchet-based activity focused on teaching analytical design. 

1.3. Transition from Seminar-Based Course 

Considering the benefits of PBL, an introductory freshman course, GE 100, was completely 
redesigned in fall 2012 to help students experience the range of technical topics covered in the 
department using a single hands-on project, and begin to develop engineering judgment. Before fall 
2012, GE 100 was a 1 h, seminar-based course consisting of overview lectures provided by department 
faculty. While students were exposed to a wealth of important information, the course on the whole 
did not engage students, and did little to prepare them for the upcoming rigorous curriculum. Each 
lecture was delivered by a different faculty member with varying levels of engagement and utility for 
first-year students. The revised course maintained the same session format: 50 min meetings every 
week for half a semester (eight weeks for a total of 6.7 h). This limited direct contact with the students 
presented an additional challenge when developing an appropriate course curriculum. 

1.4. Curriculum Overview 

In the redesigned GE 100 student project groups assemble, design, and test medium-scale 
trebuchets as part of the project (see Table 1 for the curriculum outline). Reconfigurable trebuchet 
kits were developed that supported a streamlined yet intensive hands-on experience. Student 
groups (3-5 students each) followed technical instructions to use hand tools to assemble complete 
trebuchets from components provided in kits (meetings 1 and 2). Students also learned how to adjust 
several components, such as pivot location, as a means to influence trebuchet dynamic behavior 
and projectile range. The reconfigurability of these components is a novel feature of these new kits, 
and is crucial for the use of these kits in teaching different approaches for engineering design. The 
design problem presented to students is to determine what is the best setting for four adjustable 
trebuchet components for maximizing projectile range. This is challenging to solve using intuition 
alone due to the dimension of the design problem (four correlated design variables), uncertainty in 
the trebuchet mechanisms and the operating environment (outdoors), and the significant nonlinearity 
of the overall system. 

Early in the course, GE 100 students learn basic design of experiments (DOE) techniques 
(meeting 2). With this knowledge and some initial intuition stemming from hands-on exploration 
of the trebuchets, students are then asked to plan out a set of experiments. Each experiment is a 
distinct trebuchet design (i.e., combination of settings for reconfigurable variables) that the students 
would like to test physically. The number of physical tests is limited severely because the tests are 
conducted outdoors during class time, and adjustments must be made between tests. Over the 
course of several semesters, the trebuchet kits have been refined iteratively to streamline assembly 
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Table 1. Curriculum outline. 


Meeting 

Week 

Type 

Name 

1 

1 

Hands-on 

Trebuchet Assembly 

2 

2 

Hands-on 

Trebuchet Assembly and DOE Activity 

3 

4 

Lecture 

Process Design Overview 

4 

6 

Hands-on 

Field Day #1 

5 

7 

Lecture 

Trebuchet Physics Overview 

6 

7 

Hands-on 

Model-based Design Activity 

7 

8 

Hands-on 

Field Day #2 

8 

11 

Lecture 

Project Analysis and Reflection 


and reduce adjustment time, but the time required to make adjustments still limits the total number 
of experiments that each project group can perform. During this initial set of experiments, the 
students have very limited design intuition for the trebuchets, so their experiments are often designed 
to sample the trebuchet design space broadly ( i.e ., students use a space-filling experiment design, 
see Figure 1). During this activity, a handout required students to perform the following activities 
in order: 

1. Rank the design variables from 1 to 4 in order of importance; 

2. Identify variable ranges (min and max) that you would like to test using the physical trebuchet 
kit for guidance; 

3. Plan for a budget of eight total experiments with suggestions for space-filling experiments. 
These tasks provided an introduction to the systematic process of designing experiments. 



Figure 1. Space-filling Latin Hypercube sampling plan with three variables and eight sample points [28]. 

The first lecture provides an overview of both the course and the department. An introduction 
to process modeling is also given during this first lecture (meeting 3). A logistics model was 
created that combines agent-based modeling [29] and discrete-event modeling [30] to simulate how 
student groups rotate through trebuchet tests on field days (outdoor test days). Process modeling 
helps introduce students to some of the department's courses and core research areas, provides an 
additional modeling experience beyond physics-based models, and familiarizes students with the 
field day logistics. The next meeting with the students was the first outdoor field day (meeting 4) 
where students execute tests specified during the DOE activity, following logistics policies refined 
using the model. 

After the first field day tests, students learn in class about the physics of trebuchets, the basics 
of simulating nonlinear dynamic systems, and model-based design (meeting 5). This instruction 
is followed-up with a computer modeling lab where students are introduced to a sophisticated 
physics-based, multibody dynamic trebuchet model that predicts projectile range as a function of 
trebuchet design variable values (meeting 6). During this modeling lab project groups use this model 
to test new design ideas much more rapidly than through physical experiments. Student realize 
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quickly the value of using physics-based models in engineering design. Due to the rapid response 
from the model, most student groups are able to develop intuition for the influence of design changes 
and converge on a near-optimal trebuchet design. During this activity, a handout required students 
to perform the following activities in order: 

1. Perform three predefined tests and report the distance; 

2. Answer the following question: "What is a good finger angle when sling length is 26 in and the 
pivot location is 4 in? Why?" (which is a single variable optimization problem); 

3. Rank the design variables from 1 to 4 in order of importance (repeat from the DOE activity); 

4. Plan for a budget of six total experiments for field day #2; 

5. (optional) Test more designs at home. 

A new experiment design is constructed based on the insights gained from the first field day 
experiments and the computer modeling activity. The new experimental plan is fundamentally 
different from the first; trebuchet designs to be tested are clustered around the design that was 
predicted to be optimal during the modeling lab. The purpose of the clustered experimental plan 
is to account for modeling error and improve the chances of producing the best possible range for 
a physical trebuchet test. With the new experimental plan, the students then go back out to the test 
area for a second test launch activity (meeting 7). The student groups largely experience a dramatic 
improvement in the performance of their trebuchet designs. 

The final meeting (8) was devoted to analysis and reflection on the previous seven meetings. 
This meeting included the following activities: 

1. Review of both quantitative and qualitative results from all of the project activities, including 
analysis of experimental results; 

2. Comparison of variable rankings based on student intuition (before and after the modeling lab); 

3. Discussion on the effectiveness of the launch day movement policies and comparison with 
multi-agent modeling results; 

4. Exploration of a comprehensive data set based on the trebuchet model to more fully understand 
the trebuchet design problem and the complex tradeoffs involved; 

5. Discussion on the explicit reasons for why engineering students take math and engineering 
analysis courses and how these topics are directly useful in engineering practice; 

6. Asking students their definitions of engineering intuition and judgment, and how course 
activities influenced their understanding of these ideas; 

7. Discussion on the value and potential problems with using modeling and simulation in 
engineering practice; 

8. Engaging students in a discussion on what they learned about developing a systems perspective 
for engineering design; 

9. Review of a variety of strategies for engineering design, including their different levels of 
analytical rigor, as well as the role of human creativity across all design strategies. 

Many of these points are further elaborated in the following sections. 

1.5. Learning Objectives 

This sequence of activities provides students with first-hand experience that demonstrates the 
value of mathematical and physics-based knowledge in practical engineering design activities, and 
also capitalizes on the value of hands-on experiences to deepen student learning. Students gain 
valuable insights into why it is essential for engineers to learn foundational topics in mathematics 
and science. It is our objective to provide students at the very beginning of their college education 
with a solid understanding of how fundamental analysis courses required at early stages of their 
education provide an important foundation for later design courses, and how these courses will help 
prepare them to become successful engineers. 
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In the subsequent sections, we discuss the trebuchet kits used in this curriculum followed by 
a description of computational model used to explore the design space. Lastly we present the 
perceived improvement in the students' engineering judgment outcomes due to the inclusion of this 
project-based trebuchet activity, supported by the rigorous statistical analysis. Engineering judgment 
here refers to the ability to utilize simultaneously practical engineering intuition and quantitative 
engineering analysis to support improved engineering decisions. 

2. Reconfigurable Trebuchet Kits 

In this section, we describe the design of customized trebuchet kits used by students in this 
redesigned GE 100 course. The primary motivation for designing these kits in-house was due to 
unavailability of commercial trebuchet kits that provided the adjustability needed to allow students 
to make design decisions. A new reconfigurable trebuchet kit was designed that allows continuous 
adjustments to pivot position, sling length, and finger angle (see Figure 2a). The kits also support 
one discrete design decision: whether or not wheels are used on the trebuchet base. The kits were 
designed for durability, ease of use by students (streamlined assembly and adjustments), and ease 
of fabrication (no specialized equipment required). Engineering drawings, course materials, and 
software models have been made freely available so that other institutions may use this project as 
a teaching aid. We now look briefly at some of the design aspects of these trebuchet kits. A number 
of factors shaped the final design of the trebuchet: 

• Reproducibility across trebuchet kits for fair comparisons and use with a computer model; 

• Reconfigurable capability to support rapid student adjustment of design variables (under 5 min); 

• Streamlined assembly process, enabling most student groups to complete assembly process in 
less than 40 min given written instructions, components, fasteners, and tools; 

• Portability (two college-level students can transport them small distances); 

• Durability (multiple years of use, rain/weather protection); 

• Reasonable cost (approximately $150 per kit, including all materials, hardware, tools, and 
manufacturing tools). 

A CAD drawing of an assembled trebuchet is shown in Figure 2a. During operation, the user 
places a soft clay projectile in the pouch at the end of the sling, raises the counterweight to its 
maximum vertical position, positions the projectile on the base being careful to remove sling slack, 
and then releases the counterweight to initiate projectile launch. As the weight moves down in curved 
motion it swings the sling up and forward. One end of the sling is a loop that slid over the "finger". 
The finger angle (one of the adjustments) determines the point at which the sling loop slides off the 
finger, releasing the projectile. The finger angle influences both the launch angle and velocity. Other 
adjustments also influence the point at which the sling slides off the finger. 



Figure 2. Cont. 
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Figure 2. Reconfigurable trebuchet kit. (a) CAD drawing of trebuchet illustrating the four design 
variables; (b) measuring conventions for finger angle and pivot position. 


Students can make four distinct trebuchet adjustments to influence dynamic behavior and 
projectile range. These four adjustments (design variables) are detailed in Table 2. 

Table 2. Reconfigurable design variables for trebuchet kits, with lower and upper 
bound specifications. 



Name 

Type 

LB 

UB 

*1 

Sling Length 

Continuous 

5 in 

35 in 

*2 

Finger Angle 

Continuous 

-40° 

O 

O 

*3 

Pivot Position 

Continuous 

0 in 

8 in 

X 4 

Wheels 

Discrete 

On 

Off 


The first three design variables are continuous while the fourth is discrete (the design can either 
have four wheels attached at the base or not). Variable upper and lower bounds were provided to the 
students to help them understand the range of values that could be used for the continuous design 
variables. The measuring conventions for and X 3 are shown in Figure 2b. 

The assembly steps and design variable adjustments require some basic tools. These were 
provided for each trebuchet kit in the form of a toolbox (see Figure 3(a)). This toolbox contained 
two nutdrivers, two adjustable wrenches, a hammer, a protractor, and a tape measure. The toolbox 
also served as storage for the fasteners and smaller parts. 



Figure 3. Photographs of a student toolkit and a trebuchet. (a) toolbox and select tools for assembly 
and design variable readjustments; (b) assembled trebuchet. 

Note that some of the parts are preassembled such as the shoulder (part A in Figure 4) and 
caster or wheel (part C). Students had a limited amount of time to assemble trebuchet kits in class, 
so some limited preassembly was used to reduce total assembly time. The students are provided 
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assembly instructions with both a written description and simple drawings that illustrate assembly 
steps. In addition, a mentor (graduate student or senior undergraduate student) was available if the 
students had any questions. The six assembly steps are illustrated in Figures 4 and 5. 

Each trebuchet kit has a counterbalance mass consisting of two 10 lbs plates (part D). One of 
the assembled trebuchets is shown in Figure 3(b). The trebuchet body is approximately 36 inches 
long, 11 inches wide and 22 inches tall. The throwing arm (without finger) is approximately 36 inches 
long. The trebuchet parts such as base (part B), shoulders, and throwing arm (part G) are all made of 
wood. Most wood components are pine, although high-stress components (such as the throwing arm, 
finger, and main uprights) are oak. The pivot rod (part 6), pins, and fasteners are made of steel. The 
looping clamp mechanism that simplifies sling length adjustment, and the mounting seats for casters 
(shown in Figure 6g) were both fabricated via 3D printing (polymer fusion deposition modeling). 
The 3D printed caster mounting seats streamlined assembly and reduced the time required to switch 
between with and without wheels configurations. These seats enable assembly with a single large 
central bolt instead of four machine screws. 



Figure 4. Drawings illustrating steps 1-3 in the assembly instructions. Step (1) involves fastening of 
casters to the trebuchet base. Step (2) illustrates attachment of upright shoulders to the base. Step (3) 
involves assembly of the pivot mechanism and attachment of the counterweights. 
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Figure 5. Drawings illustrating steps 3-6 in the assembly instructions. Step (4) illustrates how the 
finger assembly is to be installed. The throwing arm assembly is then attached to the trebuchet body 
via the fulcrum in Step (5). Step (6) shows how to attach the sling. The end of the sling cord with a 
loop is first slid over the finger (part F), and then the other end of the sling is threaded through the 
clamp mechanism (part E). This clamp mechanism allows rapid adjustment of sling length without 
the need to untie or tie knots, and also prevents sling cord damage. 
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(a) sling and finger angle 


(b) pivot position 



(d) sling length adjuster 




(c) removable wheels 




(f) projectile 


(g) 3D printed parts 


Figure 6. Photographs of trebuchet components. 


3. Interactive Computational Models 

In this section, we introduce the interactive computational models used in the GE 100 curriculum 
to help students learn the value of model-based design. 

3.1. Design for Maximum Range through Physics-Based Modeling 

GE 100 students launch the trebuchets on the field during the first launch activity. They choose 
their design parameters such as sling length, finger angle and pivot locations, based on intuition 
stemming from only limited experience with the trebuchets. After the first launch day, students learn 
at an introductory level the physics of trebuchets, including techniques for simulation of nonlinear 
dynamic systems. This not only helps them develop deeper physics-based intuition for the influence 
of design changes on trebuchet behavior, but it also provides a foundational understanding for how 
sophisticated computational models simulate trebuchet behavior. After learning about trebuchet 
physics, students complete a modeling lab activity where they are trained in the use of a multibody 
dynamics trebuchet model. They can then perform rapid design studies by iteratively adjusting 
design variable values and obtaining projectile range and other results through simulation. Students 
repeat this process of updating their design based on simulation results until they converge on a 
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design that they are satisfied with. In addition to helping students arrive at a significantly improved 
design, this modeling lab was also shown to help students improve qualitative understanding and 
intuition for trebuchet system design. 

The physics-based simulation model for trebuchet was developed in MATLAB® and 
SlMMECHANICS™ software. The software interface for this simulation model is shown in Figure 7. 
The goal of providing students with a computer simulation model is to help them experience directly 
the importance of model-based design in the overall design process. It helps them to develop more 
accurate intuition through rapid creative exploration of the design space, and demonstrates through 
experience the value of rapid simulations when striving to solve a design problem quickly [31], and 
to identify better designs than can be achieved through slower physical testing. 



Figure 7. Graphical user interface for simulation model. 

The simulation model is used by students to explore the design space and to discover the relative 
importance of design variables on improving range. This intuition helps students identify designs 
quickly that maximize range according to simulated results. The produces two important outputs: 
(1) predicted throwing range for a given design and the predicted path of projectile as shown in 
Figure 8; and (2) an animation of trebuchet motion during the sling launch as shown in Figure 7. This 
rich set of outputs is designed to help students gain insight into how the system behaves, and how 
design changes influence specific aspects of dynamic behavior. 

While the trebuchet system appears to be mechanically simple, its underlying equations of 
motion are highly nonlinear due to large-angle rotations of the throwing arm and sling. This system 
also involves multiple bodies with irregular inertias such as the base, arm, counterbalance weights, 
wheels, and other components. The governing differential equations for the trebuchet system cannot 
be solved in closed-form [32], so must be solved via numerical simulation. The trebuchet model 
is further complicated in the case where wheels are attached and the base moves relative to the 
ground during launch. Another complication is the sliding contact between the projectile and the 
base (i.e., where it slides before being lifted off and slung forward by the arm). The launch process 
involve three distinct phases separated by two discrete events: (1) when the projectile lifts off the base; 
and (2) when the projectile is released. This results in a hybrid dynamic system [33] with three phases. 
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Figure 2 
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Figure 8. The range plot pop-up. 


These challenges motivate the use of MATLAB®/SimMechanics™. These software tools 
simplify the process of modeling a complicated multibody dynamic system. Engineers specify the 
physical properties of the system, and then the equations of motion are constructed internally. These 
equations are then solved by numerical simulation. Even though a great deal of care is taken 
to model the trebuchet system in the MATLAB®/SimMechanics™ with as much as accuracy 
possible, certain physical aspects cannot be captured with complete accuracy—such as nonlinear 
friction between mechanical components, air resistance, and minor manufacturing variations across 
kits. We have ongoing research into these areas to improve the model through computer vision and 
system identification. 

Freshman students did not construct the trebuchet simulation model. This model was developed 
and refined a priori, and students used it as a tool to explore trebuchet designs, record data, visualize 
results, and automatically generate results and data reports. 

Detailed physics-based models are not only useful for students or engineers seeking to optimally 
tune an existing reconfigurable system (such as the trebuchet here), but they are also valuable for 
the development of new reconfigurable systems. For example, it was important in developing the 
trebuchet kits to ensure that a sufficient level (range) of reconfigurability was provided such that 
students can explore the desired range of system behavior. 

Similarly, this model can be used to identify more narrow ranges of reconfigurability (defined 
by variable bounds) with the objective of reducing design space size in a way that helps improve 
the chances that students will identify successful designs. It was found that allowing students to 
explore the entire range of trebuchet reconfigurability resulted in frustration; significant time was 
spent exploring unproductive design space regions. A balance must be struck between presenting a 
solvable problem to students and providing them with a challenging learning activity that will help 
them develop the desired intuition. 

Appropriate ranges for continuous design variables (defined by upper and lower bounds) were 
identified using the multibody dynamic model. Each of the continuous design variables was varied 
over a large range of values. This study is shown in Figure 9; individual box plots show the 
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distribution of the range varying the other two continuous design variables. The median range is 
the central white line, quartile 1 to quartile 2 is shown with a black box, and the range excluding 
outliers has a thinner black line. Outliers are marked with small red dots. From these studies, we see 
that the sling length is the most influential of the design variables since the distribution of range is 
most strongly correlated with the sling length (i.e., large median when 20 < X\ < 30). However, we 
note that the model seems to break down at around 40 inches for the sling length. Similar conclusions 
can be drawn for the other two variables. This study was used to identify appropriate values for 
variable bounds provided to the students in Table 2. 



R = 0 


H— 


outliers 


median 


h box plot 



Figure 9. Sensitivity studies exploring range for each continuous design variable. 


Looking at single design variable plots often cannot reveal coupling relationships between 
design variables. In addition, while it may be simpler to tune a design a single variable at a time, 
this process may not arrive at a desirable design in an adequate amount of time. Figure 10 reveals 
the coupling between the sling length and finger angle for two select values of the pivot position. 
This type of dependence between the range and design variables can make it challenging to arrive 
at the global optimal design via manual adjustments. Teaching freshman engineering students 
these principles using a relatively simple problem can help them develop the type of qualitative 
understanding regarding design variable interaction that is crucial in the development of more 
complex engineering systems. 



Figure 10. Coupling between finger angle and sling length for two fixed pivot position values, 
(a) X 3 — 4.1 in; (b) X 3 = 8 in. 
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3.2. Process Model for Field Day Logistics 

A logistics model was developed that combines agent-based modeling [29] and discrete-event 
modeling [30] in AnyLOGIC® to model the students behavior during the field day activities. This 
model is introduced to the students before the physics-based SlMMECHANICS™ model so that 
the students can get familiar with the field day logistics through the animation demonstrated in 
Figure lib, while learning important modeling principles and considerations, such as assumptions, 
constraints, required data, performance criterion, and protocols. 


Run 


Adjust the input parameters to see how the output data changes. 


Number of teams 

Number of stations 


Protocol 


Number of adjustments allowed 


I 

1 30 30 

I 

1 10 10 

(•) Assigned Stations 
O Random Stations 
O Shortest Queue 

- 1 - 

0 1 3 



Figure 11. Cont. 
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Figure 11. Field day logistics model built with AnyLogic® . (a) parameter page; (b) screenshot of 
live animation of the field day logistics model; (c) simulation output page. 


Agent-based modeling techniques are used to approximate the movement of students through 
the launch and staging areas during testing activities. Space limitations constrain the number of 
student groups that can launch trebuchet projectiles at a time, so many student groups must wait for 
their turn to test. A discrete-event modeling framework was used to simulate movement through 
launch queues. Queues fill up based on how the teams decide to join, directions from staff, and other 
factors. Three different protocols were considered for how the agents (student groups) would interact 
as they form queue at the launch pads: 

• Assigned stations — Each team is assigned a station and they rotate with other teams assigned 

to the same station; 

• Random stations — Teams join a random queue without any thought; 

• Shortest queue — When teams are ready, they join a station with the shortest line. 

Some other tunable parameters included the number of adjustments allowed on the launch pad 
and the number of throws per entry (see Figure 11a for the complete list of tunable parameters). 
Additional constraints considered include the number of throwing stations (i.e., queues, limited due 
to space constraints), the total amount of class time (50 min), and the limited number of projectiles. 
When all available projectiles have been launched, the the instructor halts launches and course staff 
retrieve the projectiles from the field. Some of these constraints were also made tunable so the 
students could observe their effect on simulation results. A number of different performance criterion 
available for the students to consider including total number of throws, number of throws for the 
slowest team, and the number of teams with throws below a certain threshold. Students sought 
a launch policy that increased the total number of projectile launches, while also striving to fairly 
distribute tests among project groups. 

A large number of parameter values were needed to fully realize an accurate representation of 
the AnyLOGIC® model. Model parameter estimation was performed using data obtained during a 
mock launch day conducted by student volunteers not in the class. Model parameters were estimated 
from physical tests such as time to adjust specific trebuchet design variables, total time to launch the 
projectile when on the throwing station, projectile retrieval time for a given number of projectiles and 
course staff members, and walking speed of the student groups with the trebuchets (which can be 
determined from the simulation output page in Figure 11c). Under the conditions during fall 2014, 
it is seen that the assigned station protocol results in the largest total number of throws but with 
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moderate variability in the distribution of throws between groups. The shortest queue had a slightly 
smaller total number of throws but more uniformity between the groups. The random queue was 
substantially worse than the other two. 

4. Student Improvements in Trebuchet Comprehension and Engineering Judgment 

In this section, we provide some results that highlight students' improvement in trebuchet design 
comprehension and engineering judgment throughout the course stages. These results are important 
elements of the mixed-method (triangulation) approach used to determine whether this project-based 
curriculum was effective at achieving the learning objectives [34,37]. To that end, we begin with some 
relevant background information pertaining to the collected data. 

4.1. Setup 

4.1.1. Participants 

The launch data analyzed in this section was obtained from a large group of freshmen 
engineering students (109 students were enrolled in fall 2014, which is the semester this data was 
obtained from). The students were divided into four or five member groups resulting in 23 total 
groups. The facilitators of the course, lab, and launch days consisted of the professor teaching the 
course, a graduate student TA, and a number of undergraduate and graduate student assistants 
contributing during modeling labs and launch days. 

4.1.2. Instruments and Design—Four Course Activities 

The data consists of the information obtained from four course assignments—a DOE activity, 
model-based design activity (see Section 3.1), and two field day trebuchet launches with physical 
experiments. The activities were completed in the following order: 

DOE —> field day #1 —> computer modeling lab —5- field day #2 

The DOE exercise consisted of several tasks that culminated in the construction of an experiment 
plan for the first field day. Task 1 required students to rank the design variables according to perceived 
influence on projectile range. Task 2 encouraged students to physically adjust all design variables to 
get an intuitive feel for the trebuchet design and identify the realistic variable bounds. Task 3 required 
students to practice adjusting the physical components of the trebuchet in a time-efficient manner. 
The final task, task 4, required students to strategically decide on eight sets of design variables they 
believed would provide insights and that would lead to a design with maximum launch distance. 

During field day #1, the students were tasked with performing a total of eight physical launches 
using the planned tests from the previous DOE exercise. Throughout the exercise, students made 
adjustments to the trebuchet settings to explore effects on projectile launch distance. Launch 
distances were self-reported based on visually determining the distance relative to equally-spaced 
10 ft markers. Because certain groups required more time to reconfigure trebuchet settings between 
launches, not every group completed the eight allotted launches. The total number of launches for 
field day #1 was 138. 

After field day #1 and a lecture on trebuchet physics and simulation, the model-based design 
lab activity helped students experience the value of physics-based modeling and simulation in the 
design process. Since the students had already completed a few physical experiments with the 
trebuchet on field day #1, they were able to appreciate the ability to test many more designs in 
a short time span using a physics-based computer model. Students were instructed in computer 
model use, including how to input different design variable values into the trebuchet model and how 
to run the simulation to obtain the predicted projectile trajectory and launch distance. The model 
also included sophisticated animation capabilities that helped the students visualize how exactly the 
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trebuchet moved for each candidate design. Students could move forward and backward in time 
with these animations with precise control to examine trebuchet dynamics very accurately. In the 
course of these studies students sought a design with maximum projectile range. During this process, 
they also developed deeper intuition for design tradeoffs and dynamic behavior than was obtained 
during earlier project activities. Based on this model-based design exploration activity, student 
groups were asked again to predict the relative importance of the design variables with respect to 
improving projectile range. At the end of the exercise, students used the model interface to create 
an automatically generated report that presented a summary of tested designs. A second experiment 
plan was constructed for use during field day #2. This plan was distinct from the first experiment 
plan in that designs were clustered around the predicted best design. Both the experiment plan and 
the model-based design exploration report were submitted to the instructor and were provided for 
students to use during the second field day tests. 

During field day #2, students remained in their same respective groups as the first field day. 
Each group was allotted two rounds of eight launches. Most groups performed their 16 launches, 
but there were some that could not or chose not to finish all the launches. Unlike the first field day, 
the students were strongly encouraged to explore "on the fly" different values of the design variables 
to achieve the best range possible, learning from each physical experiment. The total number of 
launches for the second field day was 180,42 more than the first field day. 

4.2. Results and Discussion 

4.2.1. Design Variable Rankings 

Table 3 shows the student and instructor rankings of the design variables. The rankings for 
students (shown in DOE activity and MBD activity) is fractional since its averaged over all the student 
groups. The student rankings of design variables in the DOE activity case is completely based 
on intuition, where-in they assign highest importance to pivot position (score 1.8). After students 
complete the day 1 launches and model-based design (MBD) activity they on average identify that the 
finger angle is most important design variable (score 1.8). Lastly, the instructor's ranking of design 
variables is based on results of comprehensive computer simulations, shown in Figures 9 and 10. 
These results suggest that the combination of experiments during field day #1 and the model-based 
design activity improved the students' intuition of the trebuchet design variables and is a direct 
quantitative measure of the learning outcomes [34]. 


Table 3. Student and instructor rankings of the design variables. 


Variable 

DOE Activity 

MBD Activity 

Instructor 

Sling Length 

2.3 

2.2 

1 

Finger Angle 

2.0 

1.8 

2 

Pivot Position 

1.8 

3.4 

4 

Wheels 

3.9 

2.5 

3 


4.2.2. Statistical Analysis of Field Day Launch Distances 

Descriptive statistics were used to build a profile for the two field day launch experiments. 
A summary of that profile is presented in Table 4. The statistics show that the mean launch 
distances for day 2 was 15.65 feet longer than that of day 1, and that the standard deviation of the 
launch distances for launch day 2 was 3.76 feet smaller than day 1. Figure 12 shows the difference 
between the launch distances between days providing a performance overview for both days. It also 
illustrates how day 2 launches were longer and less sporadic, however there were some outliers. It 
can be inferred that students were correctly adjusting the trebuchet variables to maximize launch 
distances. It is therefore hypothesized that the students' improvements of launch distance on day 
2 were due to student's increased knowledge of the complete trebuchet system learned through the 





Educ. Sci. 2016, 6, 7 


19 of 25 


previous activities. A more appropriate experimental design to test the above hypothesis would be 
to randomize the sequence the students perform the first three activities (DOE, field day #1, and 
computer modeling lab) and to test the above hypothesis related to the effect of the activities. 



Figure 12. Boxplots for launch distances for both field day #1 and field day #2. 

The maximum launch distance for day 1 was 114 feet and it was performed by group 15 on 
launch #2. The maximum launch distance for day 2 was 116 feet and it was performed by group 15 
on launch #6. Out of the total launches for both days, 77% of the maximum distances launched were 
launched on day 2. Fifty-two percent of day 1 launches traveled 90 feet or greater in comparison to 
83% of day 2. The most repeated launch distance was 80 feet which had 12 occurrences on day 1 and 
17 occurrences on day 2. 

A before and after scenario was staged to determine if the project-based curriculum supported 
students' engineering judgment. The experiment tests the influence a physics-based trebuchet model 
along with a computer simulated trebuchet launching activity has on launch distances. 

• ji\'. field day #1 launch distance (before model-based design activity); 

• }i 2 '- field day #2 launch distance (after model-based design activity and field day #1); 

• Ho : }i\ — }i 2 ', There is no difference between mean launch distances of day 1 and day 2; 

• H a : ji\ }i 2 ', There is a difference between mean launch distances of day 1 and day 2. 

A paired sample f-test was used to test the hypothesis. The analyses indicated there was a 
statistically significant difference between the mean launch distances of day 1 and day 2; p < 0.05. 
Table 4 details the full results. With the knowledge that the mean of the two launch day distances 
were significantly different, the Hq can be rejected and the H a is accepted. Since the earlier descriptive 
statistics data (in Table 4) showed day 2 having a greater mean launch distance, it can be concluded 
that there were statistically significantly greater distances launched on day 2. 

Table 4. Statistical analysis of the difference between field day #1 and #2 launch distances. 

Quantity x cr SE* 95% Cl T Df 

ARange (day 1 - day 2) 15.652 38.745 3.298 (9.130,22.174) 4.746 137 


The between day findings suggest that students were better equipped for the second field day 
launches. On the first field day, students relied heavily on nascent intuition to predict successful 
trebuchet design variable values using a space-filling strategy. The launches performed by students 
on second field day were heavily influenced by the preferred solutions they obtained through 
the simulation-based design exploration exercises. Although the individual student experiences 
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were not reported, the data clearly showed some benefit of targeted experience with engineering 
analysis and simulation-based exploration on engineering judgment through improved launch 
distance performance. Analysis of student essays, combined with quantitative data, strengthens 
evidence that this pedagogical strategy does have an impact on the ability of students to integrate 
analytical engineering knowledge with practical design tasks to improve engineering decision 
making. Students were largely able to develop an understanding of how physical experiments 
differed from model results, especially after a larger number of test launches on the second field day. 

It should be cautioned, however, that the study does not attribute these findings only to the 
model-based design because the analysis did not take into account other possible factors such as 
familiarity with the trebuchets due to field day #1 or ambient weather conditions. 

The marginal means of the launch distance between the days for each group is shown in 
Figure 13. Group #6 and #19 representing low and high mean ranges respectively. Since the study was 
most concerned with improved range, the trebuchet design for group #19 perhaps could serve as an 
important reference point for maximum distance. We also see that 20 of the 23 groups (87%) having 
greater mean distances on day 2. However, to determine whether this meant there was a significant 
difference between groups, a univariate analysis was applied (see Table 5 for a summary of this 
analysis). The analysis found that there were statistically significant differences in launch distances 
between groups, between days, and between groups and days combined. Therefore, even though 
each group was unique, their progress between days favored longer launches. This is important 
because it signals that some groups outperformed other groups. What should be investigated is how 
well students in the different groups understood the various physics, mathematics, and engineering 
concepts involved in the activity, and whether these differences in understanding correlate with 
trebuchet design performance. The two-sample t -test that was performed in the study did not identify 
which groups had significantly lower launch distances than other groups. These groups can be 
identified by using an ad hoc test or by comparing trebuchet variable data with the optimization 
variable ranking model. 



Group 


Figure 13. Marginal means of distance between the field days for each group. 
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Table 5. Univariate analysis on dependent variable launch distance. 


Source 

Type III Sum Sq. 

% 

Mean Square 

F 

P 

Corrected Model 

90972° 

45 

2022 

3.2 

0 

Intercept 

1152161 

1 

1152161 

1827.0 

0 

Group 

40067 

22 

1821 

2.9 

0 

Day 

25845 

1 

25845 

41.0 

0 

Group x Day 

23328 

22 

1060 

1.7 

0.031 

Error 

171530 

272 

631 



Total 

1537276 

318 




Corrected Total 

262503 

317 





° : R 2 = 0.347, =0.238. 


4.2.3. Qualitative Assessment of Student Reflections on Mathematical and Scientific Tools in 
Engineering Practice 

At the end of the course, the students were required to answer a few questions in short essay 
format. One of the required questions was: 

Hozv do mathematical and scientific tools, such as mathematical modeling and simulation, play an 
important role in engineering practice? 

Understanding the role of mathematical and scientific tools in engineering practice is a 
requirement for good engineering judgment. Therefore the essay responses were qualitatively 
studied to see the student's final thoughts on this concept. 

After reviewing student responses, several themes surfaced. Intuition building, process 
improvements, prototyping, and optimization were all described as uses of the mathematical and 
scientific tools in the engineering design process. Many expressed that their current and future 
courses may primarily focus on building their technical knowledge but most practical engineering 
challenges require the application of this technical knowledge. Some students acknowledged that 
their fundamental engineering courses will form a basis for their engineering intuition and will teach 
them the quantitative tools required to solve complex engineering problems. All of these responses 
are aligned with good engineering judgment. This assessment was direct qualitative measure of the 
learning outcomes [34]. 

4.2.4. Students vs. Optimization Algorithm 

Students reported each test that they performed during the model-based design activity. 
Their test numbers were then compared to the tests performed by two common optimization 
algorithms to compare their search procedure and performance. The first optimization algorithm 
was gradient-based, namely fmincon in MATLAB® [35]. The other algorithm was gradient-free, 
namely patternsearch in MATLAB® [36]. Each data point in this comparison represents a distinct 
function call, i.e., a single execution of the trebuchet simulation (whether by student or optimization 
algorithm) for a unique trebuchet design. Optimization algorithm tolerances were loosened to allow 
faster termination, producing results that were better aligned with the number of tests performed by 
the students. 

The first three designs tested by students were provided by the instructor so that students could 
gain some experience with the simulation process. The initial design was near the global minimum 
for range (see Figure 10); this corresponds to a very low-performance design. This low-performance 
design was used as the starting point for the optimization algorithms. In addition, the students were 
free to change the discrete wheels variable, while the optimization algorithm always had wheels on 
in the design (including wheels increases performance). Figure 14 compares the values at each test 
for the range and three continuous design variables. The two optimization algorithms are compared 
against three representative student project groups. 
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Figure 14. Student tests with computational model compared with optimization algorithms (both 

gradient-based and gradient-free). 

We first note that all groups eventually achieve a design near 130 ft (global optimum is 149 ft) 
even from a starting point that had a very poor range of —32 ft. Group A seems to understand 
the trebuchet design immediately with increasing range with every test until arriving near their 
best design very quickly. Group B had range progress similar to the gradient-based method, with 
consistently increasing range but with some lower values as well. The Group C test sequence was 
more chaotic, but still produced a general increase in range. The overall theme between the groups is 
their ability to greatly outperform the gradient-based methods with a lower number of tests and be 
on par with the gradient-free approach. This may be due to the students ability to use engineering 
intuition for physical systems in combination with quantitative analysis (i.e., engineering judgment), 
as opposed to being limited to quantitative understanding only (as is the case with the optimization 
algorithms). Optimization algorithms ultimately find better performing designs. It seems that, when 
student groups find a design that performs reasonably well, their design improvement stagnates. 

These results may be explained by the approach the students use to explore the design space, 
namely, it is observed from the data that student groups tend to tune a single variable at a time [17]. 
This is in contrast to the process used by optimization algorithms, which can move in non-orthogonal 
search directions. Even though optimization algorithms may require more simulation executions 
than a human-in-the-loop approach, they eventually arrive near the global optimum. When variable 
coupling is present (as is evidenced in Figure 10), convergence is often slow when using the strategy 
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of tuning a single variable at a time. Another factor here is that student groups were each given 
the same set of three sample designs to test initially. While the first design produced minimum 
range, the others were reasonable designs, and provided a good starting point for design exploration. 
Other starting points may have kept the students in the negative range for many of the tests (which 
explains why a reasonable guess was given). Further studies into this behavior may produce insights 
into how students think about design early in their engineering education and pave the way for 
education models that teach students how to combine human design intuition with quantitative tools 
(such as simulation and algorithmic optimization) to arrive at superior designs quickly. An important 
related topic is the influence that different types of hands-on and simulation-based design exploration 
activities has on enhancing design intuition. 

5. Conclusions 

The objective of the study presented here was to investigate how well a project-based curriculum: 
(1) helps first-year engineering students gain firsthand experience with the value of engineering 
analysis and mathematical models in practical engineering design; and (2) supports students in 
developing engineering judgment. A sequence of activities (both active and lecture forms) were 
created with these objectives in mind. 

A mixed-methods (triangulation) approach was used to determine whether this project-based 
curriculum was effective at achieving the learning objectives [34,37] . A combination of direct / indirect 
and quantitative/qualitative measures were analyzed. Discussion of the improved design variable 
ranks (Section 4.2.1), improved launch distance measurements throughout the stages (Section 4.2.2), 
short essay responses aligned with learning objectives (Section 4.2.3), nonrandom input decisions 
made to the simulation model (Section 4.2.4), and successful completion of the written assignments 
(DOE activity and model-based design activity) complemented and converge towards the conclusion 
that the curriculum, to some degree, addressed the learning objectives. 

The data also showed students making adjustments to the trebuchet variables that increased 
launch distances, even if these designs were notably different from the designs predicted as 
optimal using the model. This highlights the ability of students to combine knowledge gained 
through hands-on experience with quantitative investigations to strengthen engineering judgment 
and arrive as superior design solutions quickly. This insight paves the way for development of 
promising education models that teach students how to combine human design intuition with 
quantitative tools (such as simulation and algorithmic optimization) to design systems with superior 
performance. A more structured and comprehensive study in this area could lead to novel 
educational methods for improving engineering judgment of engineers at all experience levels. 
Additional assessment techniques not utilized in this study such as think-aloud protocols and 
pre/post evaluations could lead to novel insights that support successful development of these 
educational methods. Longitudinal studies could be conducted to understand how activities 
designed to improve engineering judgment early in the student's academic journey affect their 
preparedness and final outlook on the engineering discipline. This research area is identified as 
future work. 

Supplementary Materials: The following supplemental materials are made available online at http:// 
systemdesign.illinois.edu/projects/ trebuchets .php : 

1. SimMechanics™ model described in Section 3.1; 

2. AnyLogic® model described in Section 3.2; 

3. Manufacturing instructions; 

4. Assembly instructions; 

5. Launch day instructions given to the students to help them understand the logistics and safety of the 
launch day; 

6 . DOE activity assignment; 

7. Model-based design activity assignment; 

8 . CAD files of the various trebuchet parts; 

9. Short video of the events. 
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